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ABSTRACT 

Large-scale systems, such as MapReduce and Hadoop, perform ag- 
gressive materialization of intermediate job results in order to sup- 
port fault tolerance. When jobs correspond to exploratory queries 
submitted by data analysts, these materializations yield a large set 
of materialized views that typically capture common computation 
among successive queries from the same analyst, or even across 
queries of different analysts who test similar hypotheses. We pro- 
pose to treat these views as an opportunistic physical design and 
use them for the purpose of query optimization. We develop a novel 
query-rewrite algorithm that addresses the two main challenges in 
this context: how to search the large space of rewrites, and how 
to reason about views that contain UDFs (a common feature in 
large-scale data analytics). The algorithm, which provably finds 
the minimum-cost rewrite, is inspired by nearest-neighbor searches 
in non-metric spaces. We present an extensive experimental study 
on real-world datasets with a prototype data-analytics system based 
on Hive. The results demonstrate that our approach can result 
in dramatic performance improvements on complex data-analysis 
queries, reducing total execution time by an average of 61% and up 
to two orders of magnitude. 

I. INTRODUCTION 

Data analysts have the crucial task of analyzing the ever increas- 
ing volume of data that modern organizations collect and produc- 
ing actionable insights. As expected, this type of analysis is highly 
exploratory in nature and involves an iterative process: the data 
analyst starts with an initial query over the data, examines the re- 
sults, then reformulates the query and may even bring in additional 
data sources, and so on. Typically, these queries involve sophisti- 
cated, domain-specific operations that are linked to the type of data 
and the purpose of the analysis, e.g., performing sentiment analysis 
over tweets or computing the influence of each node within a large 
social network. 

Not surprisingly, MapReduce (MR), be it the original frame- 
work, its open-source incarnation Hadoop or derivative systems 
such as Pig and Hive that offer a declarative query language, has 



become a de-facto tool for this type of analysis. Besides offering 
scalability to large datasets, MR facilitates incorporating new data 
sources, as there is no need to define a schema upfront and import 
the data, and provides extensibility through a mechanism of user- 
defined function (UDFs) that can be applied on the data. Since the 
computational scope of a single MR job is limited, scientists typi- 
cally implement a query as an ensemble of MR jobs that feed data 
to each other. Quite often, such queries are written in a declarative 
query language, e.g., using HiveQL or PigLatin, and then automat- 
ically translated to a set of MR jobs. 

Despite the popularity of MR systems, query performance re- 
mains a critical issue which in turn affects directly the "speed" at 
which data analysts can test a hypothesis and converge to a con- 
clusion. Some gains can be achieved by reducing the overheads of 
MR, but the key impediment to performance is the inherent com- 
plexity of queries that ingest large datasets and span several MR 
jobs, a common class in practice. A-priori tuning, e.g., by reor- 
ganizing or preprocessing the data, is quite challenging due to the 
fluidity and uncertainty of exploratory analysis. 

In this paper, we show that it is possible to dramatically improve 
query performance by leveraging the built-in fault-tolerance mech- 
anism of MR as an opportunistic physical design. Specifically, we 
make the following observations: 

• Each MR job involves the materialization of intermediate re- 
sults (the output of mappers, the input of reducers and the 
output of reducers) for the purpose of failure recovery. More 
generally, a multi-stage job, such as one that is generated by 
Pig or Hive, will involve several such materializations. We 
refer to these materialized results as the artifacts of query ex- 
ecution and note that they are generated automatically as a 
by-product of query processing. 

• Given the evolutionary nature of data exploration, it is likely 
that each query has similarities to previous queries by the 
same analyst, and even to queries of other analysts who ex- 
amine the same data. For instance, several data analysts may 
perform sentiment analysis on a specific class of tweets (e.g., 
in a specific geographical area) but with a different hypothe- 
sis in mind. Hence, the computation performed by previous 
queries in the system, as captured in the generated artifacts, 
may be relevant for a new query. 

Thus, we propose to treat artifacts as opportunistically-created ma- 
terialized views and use them to rewrite a new query in the sys- 
tem. The opportunistic nature of our technique has several nice 
properties: the materialized views are generated as a by-product of 
query execution, i.e., without additional overhead; the set of views 



is naturally tailored to the current workload; and, given that large- 
scale analysis systems typically execute a large number of queries, 
it follows that there will be an equally large number of material- 
ized views and hence a good chance of finding a good rewrite for 
a new query. Our results with an implementation of this technique 
inside an industrial data-analytics system indicates that the savings 
in query execution time can be dramatic: a rewrite can reduce exe- 
cution time by up to two orders of magnitude. 

Rewriting a query using views is a well-studied problem in 
databases, yet its treatment in the context of MR involves a unique 
combination of technical challenges: there is a huge search space of 
rewrites due to the large number of materialized views in the oppor- 
tunistic physical design; queries can be arbitrarily complex; and, 
views and queries almost certainly involve UDFs. Unfortunately, 
previous works do not address the problem in its full generality and 
essentially ignore one or more of the previous dimensions. Recent 
methods to reuse MR computations such as ReStore [3] and MR- 
Share [12] lack a semantic understanding of the artifacts produced 
during execution and can only reuse/share cached results when ex- 
ecution plans are identical. We strongly believe that any practical 
solution has to address the query rewrite problem in its full gener- 
ality. 

Contributions. In this paper we present a novel query-rewrite 
algorithm that targets the scenario of opportunistic materialized 
views within an MR system. The algorithm employs techniques in- 
spired by spatial databases (specifically, nearest-neighbor searches 
in metric spaces [7]) in order to aggressively prune the huge space 
of candidate rewrites and generate the optimal rewrite in an effi- 
cient manner. Specifically, our contributions can be summarized as 
follows: 

• A gray-box UDF model that is simple but expressive enough 
to capture many common types of UDFs. This affords us 
a limited understanding of UDFs to enable effective reuse 
of previous results. We provide the model and the types of 
UDFs it admits in Section 3. 

• A rewriting algorithm that takes as input a query and a set 
of views and outputs the optimal rewrite. We show the al- 
gorithm is work-efficient in that it considers the minimal set 
of views necessary to find the optimal rewrite under certain 
assumptions. We describe this further in Section 5. 

• Experimental results showing our methods provide execution 
time improvements up to two orders of magnitude using real- 
world data and realistic complex queries in Section 7. The 
savings from our method are due to moving much less data, 
avoiding the high expense of re-reading data from raw logs 
when possible, and reusing/repurposing results from long- 
running computations including UDFs. 

2. PRELIMINARIES 

Here we present the architecture of our system and briefly de- 
scribe its individual components and how they interact, followed 
by our definitions. 

2.1 System Architecture 

Figure 1 provides a high level overview of the generic system 
framework we have developed. Our system integrates our query 
rewriting component with an existing query execution engine that 
is used by analysts. 

We make the following assumptions about our target system and 
the nature of the queries. First, the optimizer takes a query writ- 
ten in some declarative language and translates it into an execu- 
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tion plan that consists of MR jobs. Second, queries are expressed 
against the base data, which are large logs and queries contain 
UDFs. Third, each MR job materializes its output to the stable stor- 
age (e.g., HDFS in Hadoop). Fourth, we assume that the optimizer 
can provide cost estimates for UDFs admitted to the system. 

We extend the target execution engine by integrating our 
rewriter, presented in Section 5, as well as a materialized view 
metadata store. In order to let the rewriter communicate with the 
optimizer of the target engine, we extend the optimizer to generate 
a plan with two types of annotations on each plan node: (1) the log- 
ical expression of its computation and (2) the estimated execution 
cost. 

The rewriter uses the logical expression in the annotation when 
searching for rewrites for the output of a node. The expression 
consists of relational operators or UDFs. For each rewrite found 
during the search, the rewriter utilizes the optimizer to obtain a 
plan and estimated cost. 

During query execution, all by-products of query processing are 
retained as opportunistic materialized views, and stored in the sys- 
tem becoming part of its opportunistic physical design configura- 
tion. The materialized view metadata store contains information 
about the materialized views currently in the system such as the 
view definitions, and standard data statistics used in query opti- 
mization. 

2.2 Notations 

We use W to denote a plan generated by the query optimizer, 
represented as a DAG containing n nodes, ordered topologically. 
Each node represents an MR job. We denote the i th node of W as 
NODE;, i £ [l,n]. The plan has a single sink that computes the 
result of the query. Under the topological order assumption, this 
is NODE n . Wi is a sub-graph of W containing NODE; and all of 
its ancestor nodes. We refer to Wi as one of the rewritable targets 
of plan W. Following standard MR semantics, we assume that the 
output of each job is materialized to disk. Hence, a property of Wi 
is that it represents a materialization point in W. An outgoing edge 
from NODEfc to NODE; represents data flow from k to i. V is the 
set of all existing materialized views (MVs) in the system. 

The function COST(Wj) takes as input Wi and returns its es- 
timated cost provided by the query optimizer. CoST(NODE;) is 
the cost of the MR job at NODE;, given its input data. C0ST(W / i) 
= EvNODE feeWt COST(NODE fc ). 

We use n to denote an equivalent rewrite of target Wi using only 
views in V. r» is considered an equivalent rewrite of Wi iff it pro- 
duces an identical output to Wi given the same database instance 
D. A rewrite r* represents the minimum cost rewrite of W (i.e., 
W n ). 

Lw represents the language of the queries, which is the declar- 
ative query language used by an analyst when submitting a query 
to our system. This is also the language of the views. Lw includes 
all the relational operators in this query language, as well as all 
the UDFs admitted to the system by our UDF model (presented in 



Section 3). 

Lr is a subset of the declarative query language used by the 
analyst (i.e., Lr C Lw), and it represents the language used for 
rewriting queries. Lr includes select, project, join, group-by, and 
aggregation (SPJGA). Our system provides an interface to easily 
add UDFs to Lw, but every UDF admitted to the system should 
not be added to Lr. Although our techniques need not be changed, 
adding UDFs to Lr significantly increases the complexity of the 
search for rewrites. The implication of Li? C Lw is that it may 
not be possible to find an equivalent rewrite of a query in L w using 
Lij and the available views. 

2.3 Problem Definition 

Given these basic definitions, we introduce the problem we solve 
in this paper. 

Problem Statement. Given a plan W for an input query q, and a 
set of materialized views V, find the minimum cost rewrite r* ofW. 

Our rewrite algorithm considers views in V and the rewrite lan- 
guage Lr to search for r* . Since some views may contain UDFs, 
we require an understanding of UDFs in order for the rewriter to 
utilize those views during its search. Next we will describe our 
UDF model and then present our rewrite algorithm that solves this 
problem. 

3. UDF MODEL 

Queries in our scenario are likely to include complex analytical 
operations expressed as UDFs. In order to reuse previous compu- 
tation in our system effectively, we require a way to model UDFs 
semantically. 

Possibilities for modeling UDFs may include white, gray, or 
black-box approaches with varying levels of overhead and com- 
plexity to the system. A white box approach requires a complete 
description of the UDF such that the system understands how the 
UDF transforms the inputs. This approach has high overhead for 
an analyst when adding a new UDF to the system. A black box ap- 
proach has very low overhead for an analyst but produces an output 
that is completely opaque to the system thus may not be suitable for 
our goal of reusing results. Since UDFs can involve fairly complex 
operations on the data, our system adopts a gray-box approach that 
only captures the end-to-end transformations performed by a UDF. 
By end-to-end transformations, we imply that our model can cap- 
ture fine-grain dependencies between the input and output tuples, 
although our UDF model does not know anything more about the 
computation. This requires additional work to provide the gray- 
box model when adding a new UDF, yet allows the system to un- 
derstand the UDF's transformations in a useful way. A black-box 
model, on the other hand, can only track coarse-grain dependency 
between input and output as a whole. 

A UDF in our gray-box model is written as a composition of 
local functions. A local function refers to a function that operates 
on a single tuple or a single group of tuples. We restrict a local 
function to perform the following operations. 

1. Discard or add attributes 

2. Discard tuples by applying filters 

3. Perform grouping of tuples 

The gray-box model does not understand the nature of the trans- 
formation performed by the local functions however it understands 
the transformation afforded by each local function. The end-to-end 
transformation of the UDF can be obtained by composing the op- 
erations performed by each local function in the UDF 



Following our gray-box model, the input and output of every 
node in the plan is captured by three properties: attributes A, filters 
F, and group-by G. F is the conjunction of all filters applied to 
the input data, G is the current grouping applied, and A captures 
the schema. The end-to-end transformation of a UDF can be ex- 
pressed as the transformation of the input to the output using the 
composition of the local functions. Note that the composition cap- 
tures the semantics of the end-to-end transformation using the three 
operations, but not the actual computation, and not to describe the 
internal procedure. By combining these with grouping, our model 
can express rich UDFs, as well as relational operators such as se- 
lect, project, join, group-by and aggregation. Joins are modelled in 
the standard way in MapReduce which is to group multiple rela- 
tions on a common key (as in Pig [15]). 

3.1 UDF Example 

The following example illustrates how the model captures one 
possible implementation of a UDF called FOOD- SENTIMENT that 
uses a classifier to compute a sentiment score on users who talk 
about food items. 

Example 1. food-sentiment UDF is expressed as a 
composition of the following local functions: For each user, gather 
all of their tweets from the Twitter log; Apply a classifier that de- 
termines if a tweet refers to food or a closely-related concept, and 
then computes a food sentiment score for each user; Output only 
those users with score above a given threshold, 0.5. The represen- 
tation of this UDF in the gray-box model is as follows. Suppose 
the inputs are defined as attributes A={ userid, tweet/, filters 
F=0, group-by G=0, and the UDF name is FOOD-SENTIMENT. 
Then the output includes the new attribute score and an addi- 
tional filter, given by A' =( user id, score}, F' ={ score>0.5}, 
andG' =juseridj. 

Note from the three operations described, there is no facility to 
update attributes in place, which is reasonable given that our pri- 
mary data source is logs. Any modification to an input attribute 
must be reflected as a new attribute. When a new attribute is pro- 
duced by a UDF, its dependencies on the subset of the input are 
recorded as well as the UDF name. For instance, the meta-data de- 
scribing the new attribute score in Example 1 is: (^{userid, 
tweet}, F=0, G=0, name=FOOD- SENTIMENT). 

Any UDF with end-to-end transformations correctly captured by 
the model can be admitted to the system. We use this model to cap- 
ture UDFs such as classifiers, lat/lon extractor, sentiment analysis, 
log extractor and others as we show later in our experimental eval- 
uation. Note the model only needs to be provided the first time the 
UDF is added. This gray-box model has certain limitations. For 
instance, given two UDFs with different names but identical inputs 
and functionality, the new attributes they produce will have differ- 
ent meta-data. This deficiency can only be overcome by resorting 
to a white-box approach. 

3.2 Cost of UDF 

We now briefly describe how the optimizer costs a UDF given 
its gray-box description and its composition in terms of local func- 
tions. We assume that a mapping between each local function to 
the number of MR jobs, map and reduces phases are known to the 
optimizer. The optimizer costs a UDF by summing up the cost of 
performing the local functions to reasonably estimate cost. 

Suppose that a local function performs a set S of operations 
given by the three UDF model. Without a mapping of the opera- 
tions to the number of jobs, maps and reduce, we use the following 
weak property of a cost optimizer. 



DEFINITION 1. Non-subsumable cost property: Let S be any 
set of operations represented by the UDF model. Let Cost(5', D) 
be defined as the total cost of performing all operations in S on a 
database instance D. Let Cost(x, D) — oo for those operations 
x G S that cannot be applied on D. Then the following property 
states the cost of performing S on a database instance D is at least 
as much as performing the cheapest operation in S on D. 

Cost(S,D) > min(C0ST(a;,D),Va; G S) 

Because the lower bound is derived from the minimum opera- 
tion cost, it provides a weak lower bound. A tighter bound might 
be provided if we could use the cost of the most expensive opera- 
tion in S, i.e., max(CoST(s, D), Vx G S). The reason we must 
rely on the weak min property is because a stronger max prop- 
erty requires Cost(£",D) < Cost(5", D), where 5" C 5". This 
requirement is difficult to meet in practice. As a simple counter- 
example, suppose S contains a filter with high selectivity and a 
group-by operation with higher cost than applying the filter, when 
considering the operations independently on database D. Let S' 
contain only group-by. Suppose that applying the filter before 
group-by results in few or no tuples streamed to group-by. Then 
applying group-by can have nearly zero cost and it is plausible that 
C0ST(S",L>) > Cost(5',D). 

4. PROBLEM OVERVIEW 

Two factors make our problem challenging: (1) the complexity 
of answering queries using views, and (2) the property that Lr C 

iff- 

First, answering queries using views is a long-studied problem 
[1,6, 11] and known to be hard. When both queries and views are 
expressed in a language that only includes conjunctive queries, it is 
known to be NP-complete. A complete rewriting of a query using 
views is one that only uses views and does not use any of the base 
relations. Determining whether there is a complete rewriting of a 
query using a set of views is also NP-complete [11] for conjunctive 
queries. In this work we consider only complete rewrites. 

When searching for a rewrite of a target, existing views in V 
may only represent partial solutions [6] with respect to the target. 
Partial solutions can be repeatedly merged to produce complete so- 
lutions, which explodes the space further [2,6]. Candidate views 
are those considered by the rewriter, which includes all views in V 
and all hypothetical views obtained by merging views in V during 
the process of creating complete solutions. Finding the minimum 
cost equivalent rewrite of W requires considering merging views 
in V and operations in Lr, making the solution search space expo- 
nential both in 1) the number of views in V and 2) the size of Lr. 
Thus finding a rewrite for a single target in W is computationally 
expensive. 

Second, as noted in Section 2.2, Lr C Lw has the follow- 
ing implication that make a rewrite algorithm more challenging. 
Searching for an optimal rewrite of only W n does not suffice. This 
is because the inability to find a rewrite for W n does not mean 
that one cannot find a rewrite at a different target in W . If a 
rewrite rj is found for Wi, r n can be expressed as a rewrite for 
W n by combining ri with the remaining nodes in W indicated by 
NODEi+i • • • NODE n . Thus the search process must happen at all 
targets in W. 

A straightforward solution is to search for rewrites at all targets 
of W. It obtains the best rewrite for each target, if one exists, and 
chooses a subset of these to obtain r* using a dynamic program- 
ming approach. One drawback of this approach is that there is no 
easy way of early terminating the search at a single target. An- 
other drawback is that even with an early termination property, the 
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algorithm may search for a long time at a target (e.g., Wi) only to 
find an expensive rewrite, when it could have found a better (lower- 
cost) rewrite at an upstream target (e.g., Wi-i) more quickly. This 
is illustrated in Example 2. 

EXAMPLE 2. W contains 3 nodes, m, ri2, n-i, each with their 
individual costs as indicated, where cost of W = 13. Along- 
side each node is the space of views to consider for rewriting. 
Candidate views that result in a 
rewrite are indicated by triangles 
with the rewrite cost and those 
that fail to yield a rewrite are in- 
dicated by the empty triangles. An 
algorithm could have examined all 
the views at Wz finally identifying 
the rewrite with a cost of 12. How- 
ever, as noted, this algorithm can- 
not stop. Notice that by searching 
for rewrites of W\ and W2, the 
algorithm would find a rewrite of 

cost 2 for Wi, and of cost 1 for Wi. It could then combine these 
with the cost of 2 for the node n^, resulting in a rewrite 0/W3 with 
a total cost of 5. This is much less than the rewrite found for Wz 
with a cost of 12. Had the algorithm known about the low-cost 
rewrites at n\ and n,2, it could have searched less of the space at 
n-i. 

Now assume that, during the search for rewrites of a given tar- 
get, we can somehow obtain the lower bound of the cost of possible 
rewrites. Such information would be useful to stop the search ear- 
lier in two ways. First, if we know the lowest cost of rewrites in 
the unexplored space of a single target is higher than the cost of 
a rewrite already found, there is no use to continue searching the 
remaining space. Second, the failures and successes at finding a 
rewrite at one target can be used to inform the search at a differ- 
ent target. For example, if we already found the best rewrites for 
ni and n-i, we can stop search at 713 when all remaining rewrites 
for 713 have a cost greater than 5. Assuming that lower bounds are 
available, a challenging question then is how to best utilize such 
information to search this complex space of query rewriting. 

4.1 Outline of our Approach 

We propose a work efficient query rewriting algorithm that 
searches the space at each target ordered by a lower bound on the 
cost of a rewrite using a view. The lower bound should not require 
finding a valid rewrite, as this is computationally expensive. Thus 
we wish to utilize an alternative cost function that is easy to com- 
pute and has some desirable property with respect to the actual cost 
of a rewrite. 

We define an optimistic cost function 0PTC0ST(W / i, v) that 
takes a candidate view v and target Wi as inputs and provides a 
lower-bound on a rewrite Ti of Wi using v. rt is a rewrite of Wi 
that uses the candidate view v. The property of the lower bound is 

OPTCOST(Wi,7j) < COST(r,) 

which we will describe further in Section 6.2. For the purposes 
of discussion here we assume the existence of this function and it 
can be employed as a black-box. The use of a lower-bound cost 
is inspired by nearest neighbor finding problems in metric spaces 
where computing distances between objects can be computation- 
ally expensive, thus preferring an alternate distance function that is 
easy to compute with the desirable property that it is always less 
than or equal to the actual distance. 
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Given OptCost function, our rewrite algorithm finds the opti- 
mal rewrite r* of W by breaking the problem into two components: 

1. BfRewrite (Section 5) performs an efficient search of 
rewrites for all targets in W and outputs a globally optimal 
rewrite for W n ■ 

2. VmwFlNDER (Section 6) enumerates candidate views for a 
single target based on their potential to produce a low-cost 
rewrite of the target, and is utilized by BfRe WRITE. 

The VlEWFlNDER orders candidate views at a target as il- 
lustrated in the example alongside. These graphics show can- 
didate views as triangles arranged by their OptCost to a tar- 
get as indicated by the scaled line below the views. The 
OptCost of all views can be obtained quickly, which deter- 
mines their placement along the line. The top graphic shows 
the initial state of VlEWFlNDER before searching for rewrites. 
The bottom graphic shows the state of VlEWFlNDER after ex- 
amining two candidate views for rewrite. The cost of the 
rewrite for each candidate view is indicated inside the triangle. 
The VlEWFlNDER began by exam- 
ining the view at OptCost of 1 as 
this view had the potential to pro- 
duce the lowest-cost rewrite of the 
target. It obtains the actual cost of a 
rewrite using this view, found to be 6 
as marked. The next candidate view 
was at OPTCOST of 3, which is ex- 
amined and its rewrite cost was found to be 5. So, the best rewrite 
found for the target is now 5. The utility of the lower bound cost can 
be immediately seen now as there is no reason to examine the next 
candidate view which has an OptCost of 7. Thus any remaining 
views can be pruned from consideration, and the VlEWFlNDER at 
this target can terminate early. 

The BFREWRITE spawns one instance of VlEWFlNDER per tar- 
get in W and uses the information from the search at a target 
to inform and control the search at other targets. By exploiting 
OptCost our algorithm performs an incremental search with a 
VlEWFlNDER at each target. It starts with the candidate views 
at different targets that have the potential to form the lowest cost 
rewrite of W and continues its search in an incremental fashion un- 
til termination. In this way our approach obviates the need to look 
in sub-spaces where it is not possible to find a cheaper rewrite for 
W. This forms the best-first nature of our solution. Note that due 
to the known hardness of the problem, in the worst case it must 
examine all candidate views at each target. 

To illustrate BfRewrite process, the example shown alongside 
again shows W containing 3 targets m, n 2 and 713 with cost 6, 5, 
and 2, respectively. The candidate views at each target are ordered 
by their OptCost value as described previously. 

Our best-first algorithm would first examine a rewrite com- 
posed of the next candidate view at ni (OptCost of 1), the 
next candidate view at n 2 (OptCost of 2) and node n 3 (COST 
of 2). This combination has the potential to produce a rewrite 
of W3 with an OptCost of 1+2+2=5. However, upon ob- 
taining the actual rewrite of the candidate views at m and 
712, we can see that the actual cost of this rewrite is 10. 
The algorithm contin- 
ues by examining the 
next candidate view at 
m (OptCost of 3). 
This can potentially 
produce a rewrite of 
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W3 by compos- 
ing: OptCost of 3 at 
ni, plus rewrite with 

COST 4 at n 2 , plus n 3 with Cost 2 (3+4+2=9). However, this 
candidate view at m was found to have an actual cost of 6. Hence, 
this rewrite of W has a COST of 12 (6+4+2) and can be discarded 
since BfRewrite already has found a rewrite of W with COST 10. 
The algorithm need not examine any remaining candidate views in 
m and n a as they do not have the potential to produce a cheaper 
rewrite. 

5. BEST-FIRST REWRITE 

The BfRewrite algorithm produces a rewrite r of W that can 
be composed of rewrites found at multiple targets in W. The com- 
puted rewrite r* has provably the minimum cost among all pos- 
sible rewrites in the same class. Moreover, the algorithm is work- 
efficient: even though C0ST(r* ) is not known a-priori, it will never 
examine any candidate view with OptCost higher than the opti- 
mal cost CoST(r*). Intuitively, the algorithm explores only the 
part of the search space that is needed to provably find the optimal 
rewrite. 

The algorithm begins with W itself being the best rewrite for 
the plan. It then spawns n concurrent search problems at each of 
the targets in W and works in iterations to find a better rewrite. In 
each iteration, the algorithm chooses one target Wi and examines 
a candidate view at Wi. The algorithm makes use of the result of 
this step to aid in pruning the search space of other targets in W. 
To be work efficient, the algorithm must choose correctly the next 
candidate view to examine. As we will show below, the OptCost 
functionality plays an essential role in choosing the next target to 
refine. 

The BfRewrite uses an instance of the VlEWFlNDER to search 
the space of rewrites at each target. We will describe the details 
of VlEWFlNDER in Section 6. In this section, VlEWFlNDER is a 
black box that provides the following functions: (1) INIT creates 
the search space of candidate views ordered by their OptCost, (2) 
PEEK provides the OptCost of the next candidate view, and (3) 
REFINE tries to find a rewrite of the target using the next candidate 
view. One important property of REFINE is the following: there are 
no remaining rewrites to be found for the corresponding target that 
have a cost less than the value of PEEK. 

5.1 Algorithm 

In this section, we present the details of our BfRe WRITE al- 
gorithm and in the following section we provide an example that 
illustrates how the algorithm works on a small instance of the prob- 
lem. 

Algorithm 1 Optimal rewrite of W using VlEWFlNDER 

1: function BfRe write(W, V) 
2: for each Wi 6 W do 

3: VlEWFlNDER.lNIT(Wi, V) 

4: BSTPLNj +-Wj 

5: bstPlnCst^ <-CosT(Wi) 

6: end for 

7: repeat 

8: (Wi,d) +- FlNDNEXTMlNTARGETfWn) 

9: REFINETARGET(Wi) if Wi £ NULL 

10: until Wi = NULL 

1 1 : Rewrite W using BSTPLN n 

12: end function 
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> Init Step 

> original plan to produce Wi 
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Algorithm 1 presents the main BfRewrite function, which pri- 
marily repeats the following procedure: Choose the best target Wi 



C0ST=2 



(by FindNextMinTarget) and refine it by asking VmwFlNDER 
to examine the next candidate view (using REFINETARGET). 

For each target Wi £ W , the algorithm maintains the best 
rewrite BSTPLNi of Wi found so far and its cost BSTPLNCSTi 
(which are initialized in lines 2-6). The main loop iterates the 
search procedure until there is no target that can possibly improve 
BSTPLN n , at which point r* has been identified. 

In line 8, the return value Wi of FindNextMinTarget corre- 
sponds to the next target to continue searching, while d represents 
the minimum OptCost of a rewrite for W n involving a candi- 
date view at Wi that has not been examined so far. As we will see 
shortly, FindNextMinTargetQ examines views in increasing 
OptCost order at each target and so can guarantee that the return 
value d can never decrease. This property ensures that BfRewrite 
has examined all possible rewrites for W with actual cost less than 
d, which in turn allows for early termination when the cost of the 
currently best rewrite is not greater than d. 

Algorithm 2 Find next min target to refine 



Algorithm 3 Queries VlEWFlNDER in best-first manner 
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function FindNextMinTargetCW^) 
d' 4- 0; W M in NULL; d M iN «— oo 
for each incoming vertex NODEj of NODE; do 

(W k ,d) ^FindNextMinTarget^) 

d' 4- d' + d 

itdMiN > d and W k ^ NULL then 

Wmin *- W k 

dMiN 4- d 
end if 
end for 

d! 4-d! + Cost(node 4 ) 

di 4- VlEWFlNDER.PEEK() 

if min(d', di) > BSTPLNCSTi then 

return (NULL, BSTPLNCSTi) 
else if d' < di then 

return (W M in, d') 
else 

return (Wi, di) 
end if 
end function 



FindNextMinTarget, given in Algorithm 2, identifies the 
next target to be refined in W, as well as the minimum cost 
(OptCost) of a possible rewrite for Wi. There can be three out- 
come of a search at a target Wi. Case 1: Wi and all its ances- 
tors cannot provide a better rewrite. Case 2: An ancestor target of 
Wi can provide a better rewrite. Case 3: Wi can provide a bet- 
ter rewrite. By recursively making the above determination at each 
target in W, the algorithm identifies the target to refine next. 

The algorithm finds the best rewrite obtained by combin- 
ing rewrites found at the ancestors of Wi. This rewrite 
has an OptCost of d! , which is acquired by summing the 
VlEWFlNDER.PEEK values at the ancestors of Wi using the re- 
cursive procedure and the cost of NODE 4 (lines 3-11). Note that we 
also record the target Wmin representing the ancestor target with 
the minimum OptCost candidate view (lines 6-9). Next, we as- 
sign di to the next candidate view at Wi using VlEWFlNDER.PEEK 
(line 12). 

Now the algorithm deals with the three cases outlined above. If 
both d! and di are greater than or equal to BSTPLNCSTi (case 1), 
there is no need to search any further at Wi (line 13). If d' is less 
than di (line 15), then Wmin is the next target to refine (case 2). 
Else (line 18), Wi is the next target to refine (case 3). 

Algorithm 3 describes the process of refining a target Wi. Re- 
finement is a two-step process. First it obtains a rewrite n of Wi 
from VlEWFlNDER if one exists (line 2). The cost of the rewrite n 
obtained by REFINETARGET is compared against the best rewrite 
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function RefineTarget( Wi) 

Ti 4 — VlEWFlNDER. REFINE(Wi) 

if n ^ NULL and CoST(r-i) < BSTPLNCSTi then 

BSTPLNi 4- Ti 
BSTPLNCSTj <-COST(ri) 

for each edge (node 4 , NODE fe ) do 

PropBstRewrite(node^) 
end for 
end if 
end function 



function PropBstRewrite(node 4 ) 
Ti 4— plan initialized to NODEi 
for each edge (NODEj , NODEi) do 

Add BSTPLNj to r< 
end for 

if CoST(ri) < BSTPLNCsTi then 

BSTPLNCSTi <-COST(ri) 
BSTPLNi 4-Ti 

for each edge (NODEi, NODE fc ) do 

PROPBSTREWRITE(NODEfc) 

end for 
end if 
end function 



found so far at Wi. If Ti is found to be cheaper, the algorithm suit- 
ably updates BSTPLNi and BSTPLNCSTi (lines 3-9). In the second 
step (line 7), it tries to use r t as part of a rewrite of W using the 
recursive function given by PropBstRewrite in Algorithm 3. 
After this two-step refinement process, BSTPLN n contains the best 
rewrite of W found so far. 

The recursion procedure given by PropBstRewrite pushes 
downward the new BSTPLNi along the outgoing nodes and towards 
NODE n . At each step it composes a rewrite n using the immediate 
ancestor nodes of NODEi (lines 2-5). It compares n with BSTPLNi 
and updates BSTPLNi if U is found to be cheaper (lines 6-12). 

5.2 Algorithm Example 

Example 3 shows how BfRewrite attempts to rewrite a sample 
plan. 




COST=2 



Figure 2: Three targets m , n 2 and n 3 with candidate views or- 
dered by their OptCost a-f. 



EXAMPLE 3. Figure 2 shows a plan containing three nodes 
m, ri2 and nz, where n 3 represents W n - The five candidate views 
a-f are arranged by their OPTCoST/rara the target nodes. For ex- 
ample, b is placed at OPTCOST of 6 for node n 3 . The cost of nodes 
m,ri2,7i3 is 6, 5, 2 respectively, hence BSTPLN3 (;'.e.,BSTPLN„) 
begins with C0ST=i5. 

In the first step of the algorithm, the cheapest potential rewrite 
form is determined to be (a = l) + (c = 2) + (n 3 = 2) having a to- 
tal OptCost of 5, whereas the next cheapest possible rewrite ofn.3 
using b has an OptCost of 6. FindNextMinTarget chooses 
a as the first candidate view to REFINE within a + c + 713 be- 
cause the OPTCOST of a (=1) is less than c (=2). After refining 
a, the actual COST of a is found to be 4, which is shown by the 
dashed line from a in Figure 2. Therefore BSTPLNCSTi is set to 



4. Now since the best known rewrite for 713 is (a = 4) + (712 = 
5) + (ri3 = 2) is 11, the value BSTPLNCST3 is updated from 13 to 
11 by PROPBSTREWRITE. 

Next it attempts the rewrite for 713 using b with an OptCost of 6, 
which is less than the next best choice of (d = 3) + (c = 2) + (713 = 
2) with a total OptCost of 7. The COST of the rewrite ofriz using 
b was found to be 12 as indicated in Figure 2. Since BSTPLNCST3 
is already 11, it is not updated. 

The next best choice for n.i is the rewrite with the OptCost of 
7. Within that rewrite, FindNextMinTarget chooses to refine 
c, yielding a rewrite for 712 whose actual cost is 4, so BSTPLNCST2 
is set to 4. Then BSTPLN3 and BSTPLNCST3 are set to be (a = 
4) + (c = 4) + (ri3 = 2) with a total cost of 10. Figure 2 captures 
this present state of the algorithm after the first three rewrite at- 
tempts. 

The algorithm would proceed to the next best possible rewrite of 
(d = 3) + (c = 4) + (n 3 = 2) with OptCost of 9 which is still 
better than the best known rewrite 0/713 with cost of 10. The al- 
gorithm terminates when there are no possible rewrites remaining 
for W n with a OptCost less than BSTPlnCst„. Any view with 
an OptCost greater than their target node's BSTPlnCst can be 
pruned away, e.g., e at n 3 (since 11 > 10,) and f at n\ (since 
5 > 4j. 

It is noteworthy in Example 3 that had the algorithm started at 
713 first, it would have examined all the candidate views of 713, re- 
sulting in a larger search space than necessary. 

5.3 Proof of Correctness 

The following theorem provides the proof of correctness and the 
work efficiency property of our BfRewrite algorithm. 

Theorem 1. The BfRewrite finds the optimal rewrite r* of 
W and is work efficient. 

PROOF. Finding the optimal rewrite requires that the algorithm 
must not terminate before finding r*. Ensuring work efficiency re- 
quires that the algorithm should not examine any candidate views 
that cannot be possibly included in r*. First we show these prop- 
erties for a W containing a single target, then provide an outline 
of how to extend these results to an arbitrary plan containing n tar- 
gets. For the single target case, the algorithm must examine every 
candidate view with OptCost less than or equal to CoST(r* ), but 
must not examine any candidate view with OptCost greater than 
C0ST(r*). 

Proof by contradiction is as follows. Suppose that the algo- 
rithm found a candidate view v, which resulted in a rewrite r. 
Suppose that the candidate view v* which produces the optimal 
rewrite r* is not considered by the algorithm before terminating 
thus incorrectly reporting r as the optimal rewrite even though 
C0ST(r*) < C0ST(r). As the candidate view v was exam- 
ined before v* from the PQ, it must have been the case that 
OptCost(tj) < OptCost(tj*). Once v was found BSTPlnCst„ 
is set to CoST(r), which means the algorithm will not terminate 
until all candidate views whose OptCost is less than or equal to 
BSTPlnCst„ (i.e., C0ST(r)) have been found. Combining the 
above inequalities with the lower bound property of OptCost with 
respect to COST, we have: 

OptCost(tj) < OptCost(«*) < COST(r*) < Cost(t-) = 
bstPlnCst„. 

This means that the algorithm must have examined v* (and con- 
sequently r*) before terminating. Hence the contradiction that r 
was reported as the best rewrite. 

Consider the general case of W containing n targets. It suffices 
to show that the algorithm, using FindNextMinTarget, reduces 



the n priority queues to an equivalent problem of a single global 
priority queue. The queue contains all possible rewrites that include 
all candidate views from every target, and each rewrite is ordered 
by its OptCost. This reduction is straightforward and omitted due 
to lack of space. □ 

6. VIEWFINDER 

The key feature of VIEWFINDER is its OptCost functional- 
ity which is used by the BfRewrite to explore the space in an 
incremental manner and prune unnecessary sub-spaces as shown 
in Section 4.1. As noted earlier, rewriting queries using views is 
known to be a hard problem. Traditionally, methods for rewriting 
queries using views for the class of SPJG queries use a two stage 
approach [2,6]. The prune stage determines which views are rele- 
vant to the query, and among the relevant views those that contain 
all the required join predicates are termed as complete, otherwise 
they are called partial solutions. This is typically followed by a 
merge stage that joins the partial solutions using all possible equi- 
join methods on all join orders to form additional relevant views. 
The algorithm repeats until only those views that are useful for an- 
swering the query remain. 

We take a similar approach in that we identify partial and com- 
plete solutions, then follow with a merge phase. The VIEWFINDER 
considers candidate views C when searching for rewrite of a tar- 
get. C includes views in V as well as views formed by "merging" 
views in V using a MERGE function, which is an implementation 
of a standard view-merging procedure (e.g., [2,6]). Traditional ap- 
proaches merge all partial solutions to create complete solutions, 
and continues until no partial solutions remain. This "explodes" 
the space of candidate views exponentially. Our approach allows 
for a gradual explosion of the space as needed, which results in far 
fewer candidates views from being considered. 

With no early termination condition existing approaches would 
have explore the space exhaustively at all targets. Thus we desire a 
rewriting algorithm that can enumerate the space and incrementally 
explore only as much as required, frequently stopping and resum- 
ing the search as requested by BfRewrite. We note that while 
an equivalent rewrite for a target may exist, the VIEWFINDER may 
never be asked to find it, as illustrated by Example 2 for the case of 
the candidate view with cost 12 at 713. 

6.1 Algorithm 

The VIEWFINDER is presented in Algorithm 4. At a high level, 
the VIEWFINDER is stateful which enables the BfRewrite to 
start, stop and resume the incremental searches at each target. The 
VIEWFINDER maintains state using a priority queue of candidate 
views. The VIEWFINDER implements three functions INIT, PEEK 
and REFINE which we describe next. 

The INIT function instantiates an instance of the VIEWFINDER 
with a query which a logical representation of a target Wi G W 
and a set of materialized views V present in the system. Next, 
query is assigned to q and each view in V is added to priority 
queue using OptCost((j, v) as the sorting key. At the end of INIT, 
the candidate views in PQ includes only those views in V. 

The PEEK function is used by BfRewrite to obtain the 
OPTCOST of the head item in the PQ. 

The REFINE function is invoked when BfRewrite asks the 
VIEWFINDER to examine the next candidate view. At this 
stage, the VIEWFINDER pops the head item v out of PQ. The 
VIEWFINDER then generates a set of new candidate views M by 
merging v with previously popped candidate views (i.e., views in 
Seen), thereby incrementally exploding the space of candidate 
views. Note that Seen contains candidate views that have an 



OptCost less than or equal to that of v. M only retains those 
candidate that are not already in Seen, which are then inserted into 
PQ. A property of OptCost provided as a theorem later is that the 
candidate views in M have an OptCost that is greater than that 
of v and hence none of these views should have been examined be- 
fore v. This property enables a gradual explosion of the space of 
candidate views. Then, v is added to Seen. 

If v is guessed to be complete (described in Section 6.1.1), we 
try to find rewrites of q using v by invoking the RewriteEnum 
function (described in Section 6. 1.2). Among the rewrites found by 
REWRITEENUM, the cheapest rewrite is returned to BfRewrite 
as the result. 

6.1.1 Determining Partial or Complete solutions 

To determine if a view v is partial or complete with respect to a 
query q, we take an optimistic approach. This approach represents 
a guess that a complete rewrite exists using v. A guess requires the 
following necessary conditions as described in [6] (SPJ) and [4] 
(SPJG) that a view must satisfy to participate in a rewrite of q, 
although these conditions are not sufficient to confirm the existence 
of an equivalent rewrite using v. 

(i) v contains all the attributes required by q; or contains 
all the necessary attributes to produce those attributes 
in q that are not in v 

(ii) v contains weaker selection predicates than q 

(iii) v is less aggregated than q 

The function GUESSCOMPLETE(g, v) performs these checks 
and returns true if v satisfies the properties i — iii with respect to 
q. Note these conditions under-specify the requirements for deter- 
mining that a valid rewrite exists, thus a guess may result in a false 
positive, but will never result in a false negative. 



Algorithm 4 ViewFinder 



I: 


function lNIT(query, V) 


2: 


Priority Queue PQ <-0; Seen ^0; Query q 


3: 


q ^query 


4: 


for each v G V do 


5: 


PQ.add(v, OptCost((J,u)) 


6: 


end for 


7: 


end function 


I: 


function Peek 


2: 


if PQ is not empty return PQ.peekQ. OptCost else oo 


3: 


end function 


1: 


function Refine 


2: 


if not PQ.emptyO then 


3: 


v <-PQ.pop() 


4: 


M <— MERGE(u, Seen) > Discard from M those in Seen n M 


5: 


for each v' S M do 


6: 


PQ.addtV, OptCost(<j, v')) 


7: 


end for 


8: 


Seen, add(u) 


9: 


if GuessComplete(<j, v) then 


10: 


return RewriteEnum (q, v) 


11: 


end if 


12: 


end if 


13: 


return null 


14: 


end function 



6.1.2 Rewrite Enumeration 

In our system the RewriteEnum algorithm attempts to pro- 
duce a valid rewrite of a query using a view that is guessed to be 
complete. The rewrite returned represents the cheapest among all 



possible equivalent rewrites of q using v. The cost of a rewrite is 
evaluated by the COST function, and corresponds to the cheapest 
execution plan that implements the rewrite. Equivalence is deter- 
mined by ensuring that the rewrite and query contain the same at- 
tributes, filters, and group-by. 

We enumerate equivalent rewrites of q by applying compensa- 
tions [20] to a guessed to be complete view v using Lr. We do this 
by generating all permutations of required compensations and test- 
ing for equivalence, which amounts to a brute force enumeration 
of all possible rewrites given Lr. This makes case for the sys- 
tem to keep \Lr\ small. When Lr is restricted to a known, fixed 
set of operators it may suffice to examine a polynomial number of 
rewrites attempts, as in [5] for the specific case of simple aggrega- 
tions involving group-by s. Such approaches are not applicable to 
our case as the system should have the flexibility of extending Lr 
with UDFs from Lw when it results in overall system benefit. 

Given the computational cost of finding valid rewrites, 
BfRewrite limits the invocation of REWRITEENUM algo- 
rithm using two strategies. First, we avoid having to apply 
RewriteEnum on every candidate view making a guess for the 
completeness of a view based on the three properties described ear- 
lier. Second, we delay the application of REWRITEENUM to ev- 
ery complete view by determining a lower bound on the cost of a 
rewrite using v should one exist. For the lower bound we use the 
OptCost, which is described in the next section. 

6.2 Determining OptCost 

The utility of a lower bound we develop in this section enables 
the enumeration of the candidate views based their potential to pro- 
vide a low cost rewrite. OptCost relies on the non-subsumable 
cost property of the COST function to arrive at a lower-bound. 

Given that v is guessed to be complete with respect to q, a set 
difference between the attributes, filters and group-bys representa- 
tion of q and v is referred to as the fix. Fix denotes a hypothet- 
ical local function that can transform v's representation into q's. 
Note that a UDF containing such a local function may not really 
exist. We have to invoke REWRITEENUM which produces a rewrite 
containing compensations from Lr. The composition of the local 
functions in the compensation transforms v's representation to q. 
Finally, note that the existence of fix guarantee that v will result 
in a valid rewrite for the same reason that guessed to be complete 
can result in a false positive. Both assume that the required com- 
pensation operations can be applied independently of each other to 
v. 

We now describe a OptCost function with the two proper- 
ties that it is a lower bound on the cost of any plan returned by 
RewriteEnum(q, v) and inexpensive to compute. If v is a mate- 
rialized view then ci is equal to the cost of accessing v. Otherwise, 
if v results from the merging of views, then c\ is the total cost 
to access the constituent views of v. We denote C2 as the cost of 
merging the constituent views in v (i.e., creation cost) if v is al- 
ready not materialized, else C2 = if it is already materialized. We 
denote C3 as the cost of applying the least expensive operation in 
the fix on v, obtained by invoking the COST to obtain the cost of 
performing each of the operations in the fix on v. C3 is obtained by 
min(C0ST(:r, v)) such that x is an operation infix. 

The OPTCOST of v with respect to q is given by: c = ci + c 2 + 
C3, where c is less than the cost of any plan of the rewrite using v. 
If v is partial with respect to q, then C3 = since no compensation 
should be applied. 

Suppose that the optimizer can generate plans where some of 
compensations can be pushed into the candidate view v before ma- 
terializing it. In this case, OptCost can provide a weaker lower 



bound as it can only consider the cost (ci) of accessing all the con- 
stituent views of v plus the minimum cost c' 3 of applying the least 
expensive operation in the fix on any constituent views of v or on 
any intermediate view that can be created in the process of creating 
v. If v is partial with respect to q, then OptCost only includes ci. 

We provide the following proof sketch to capture the correctness 
of the OptCost function. 

THEOREM 2. c is a lower-bound on on the cost of any plan 
yielded by an equivalent rewrite r of q using v if one exists. 

PROOF. Any plan of the rewrite r will have to access all the 
constituent views in v and materialize it. So, ci and c 2 is common 
to OptCost and the cost of any plan of r. 

In order to find the lowest cost r, RewriteEnum applies all 
permutations of compensation operations to achieve an equivalent 
rewrite. Regardless of how many operations are used in the com- 
pensation, by Definition 1 , the cost of applying the compensations 
has to be at least as expensive as the cheapest operation C3 in the 
fix. 

Next we consider the OptCost function for the case involving 
push-down of compensations. For this case, both the ordering of 
the merges of the constituent views of v as well as the applica- 
ble compensations are as yet unknown. Our lower-bound holds as 
it does not make any assumptions about the ordering of the con- 
stituent views in v (i.e., by using ci) as well as the position of any 
compensation operator in any plan of r (i.e., c' 3 ) . □ 

The following theorem describes the OptCost property for 
newly merged candidates in M. This enables Algorithm 4 to gener- 
ate candidate views as needed to avoid a pre-explosion of the space 
of all candidates. 

Theorem 3. The OptCost of every candidate view in M that 
is not in Seen is greater than or equal to the OptCost of v. 

Proof. The proof sketch is as follows. The theorem is trivially 
true for v G V as all candidate views in M cannot be in Seen and 
have OptCost greater than v. If v ^ V, it is sufficient to point out 
that all constituent views of v are already in Seen since they must 
have had OptCost lesser or equal to v Hence all candidate views 
in M with OptCost smaller than v are already in Seen, and those 
with OptCost greater than v will be added to PQ if they are not 
already in PQ. □ 

7. EXPERIMENTAL EVALUATION 

In this section, we present an experimental study we conducted 
in order to validate the effectiveness of BfRewrite in finding low- 
cost rewrites of complex queries. We first evaluate our methods in 
two scenarios. The query evolution scenario (Section 7.2.1) rep- 
resents a user iteratively refining queries within a single session. 
This scenario evaluates the benefit that each new query version re- 
ceives from the opportunistic views created by previous versions 
of the query. The user evolution scenario (Section 7.2.2) repre- 
sents a new user entering the system presenting a new query. This 
scenario evaluates the benefit a new query receives from the oppor- 
tunistic views created by queries of other users. We compare the 
performance of our algorithm for the user evolution scenario (Sec- 
tion 7.2.3) with a baseline dynamic programming approach that 
searches at all targets without using OptCost. Next, we evaluate 
the scalability (Section 7.2.3) of our rewrite algorithm in compari- 
son to the dynamic programming approach. We then compare our 
method to cache-based methods (Section 7.2.4) that can only reuse 
identical previous results. We show the performance of our method 



(Section 7.2.5) under a storage reclamation policy that drops oppor- 
tunistic views. Lastly, as a sanity check (Section 7.2.6) we compare 
the quality of rewrites produced by our algorithm with a state-of- 
the-art DBMS. 

7.1 Methodology 

Our experimental system consists of 20 machines running 
Hadoop. We use HiveQL as the declarative query language, and 
Oozie as a job coordinator. UDFs are implemented in Java, Perl, 
and Python and executed using the HiveCLI. Some UDFs imple- 
mented in our system are log parser/extractor, text sentiment clas- 
sifier, sentence tokenizer, lat/lon extractor, word count, restaurant 
menu similarity, and geographical tiling, among others. 

Hive currently lacks a what-if optimizer functionality, which is 
needed to obtain cost estimates for hypothetical plans. For these 
reasons, we created a what-if optimizer that uses the cost model 
developed for MR frameworks in [14] that considers basic data 
statistics, number of map and reduce tasks, and number of jobs. 

Our experiments use the following three real-world datasets to- 
taling over 1TB: a Twitter log containing 800GB of tweets, a 
Foursquare log containing 250GB of user check-ins, and a Land- 
mark log containing 7GB of 5 million landmarks including their 
locations. The identity of a social network user (use r_i d) is com- 
mon across the Twitter and Foursquare logs, while the identity of a 
landmark (location_id) is common across the Foursquare and 
Landmarks logs. 

The metrics we report during evaluation of query performance 
are execution time in seconds and the amount of data manipu- 
lated (read/write/shuffle) in GB. When comparing BfRe WRITE to 
the Dynamic Programming algorithm, we use the following met- 
rics: running time to find the rewrite, the number of candidate 
views examined during the search for rewrites, and the number of 
valid rewrites produced. We limit both algorithms from considering 
more than 4-way joins simply for practical reasons. 

7.1.1 Evolutionary Query Workload 

Our test workload simulates 8 analysts who write complex 
queries containing UDFs, representing realistic marketing scenar- 
ios for restaurants. Each analyst in our scenario creates 4 versions 
of a query, representing the query's evolutions during data explo- 
ration and hypothesis testing. We use these 32 queries for all of 
our evaluations. On average, the original plan of a query creates 17 
opportunistic materialized views. 

We give a high-level description of these queries in Table 1 . We 
modeled our analyst queries after observing query evolutions in 
three domains - Yahoo! Pipes queries using public web data, Tav- 
erna [18] scientific workflows using biological science data, and 
TPC-DS [19] interactive queries using warehouse data. Common 
changes between versions that we identified include a parameter 
change to allow for more or less data in the results, adding a new 
subgoal to the query, adding a new UDF or replacing operations 
with a specialized UDF, and incorporating additional data sources 
to obtain richer results. Each of our query versions are revised 
in this manner, and each revision includes at least two types of 
changes. 

In the following set of experiments, AiVj corresponds to the An- 
alyst i running version j of her query. We describe one Analyst's 
query in more depth using Example 4. 

EXAMPLE 4. A2V2: Find users with affluent friends that tweet 
positively about food, and visit a lot of pubs. 

(a): EXTRACT from Foursquare log. EXTRACT restaurant name 
from text. FILTER by check-ins to wine places. GROUPBY user, 



Table 1: Eight analyst marketing scenarios used in evaluation 

Analyst 1 wants to find influential users who are interested in food for an advertisement campaign. The evolution of this scenario includes 
increasingly sophisticated interpretations of what it means to be "influential". 

Analyst2 wants to identity 100 "wine lovers" to send them a coupon for a new wine being introduced. This evolution investigates ways of 
finding suitable users to whom sending a coupon would have the most impact. 

Analyst3 wants to start a gift recommendation service where friends can send a gift certificate to a user u. We want to generate a few restaurant 
choices based on w's preferences. The evolution in this scenario will investigate how to generate a diverse set of recommendations that would 
cater to u and his close set of friends. 

Analyst4 wants to identify a good area to locate a sports bar. The area must have a lot of people who like sports and check-in to bars, but the 
area does not already have too many sports bars in relation to other areas. The evolution focuses on identifying a suitable area where there is 
good interest but low density of sports bars. 

Analyst5 wants to give restaurant owners a customer poaching tool. For each restaurant r, we identify customers who go to a "similar" restaurant 
in the area but do not visit r. The owner of r may use this to target advertisements. The evolutionary nature focuses on determining "similar" 
restaurants and their users. 

Analyst6 tries to find out if restaurants are losing loyal customers. He wants to identify those customers who used to visit more frequently but 
are now visiting other restaurants in the area so that he can send them a coupon to win them back. The evolutionary nature of this scenario will 
focus on how to identify prior active users. 

Analyst7 wants to identify the direct competition for each restaurant. He first tries to determine if there is a more successful restaurant of similar 
type in the same area. The evolutionary nature focuses on identifying what customers like about the menu, food, service, etc. that makes these 
restaurants successful. 

Analyst8 wants to recommend a high-end hotel vacation in an area users will like based on their known preferences for restaurants, theaters, 
and luxury items. The evolutionary nature focuses on matching user's preferences with the types of businesses in an area. 




Figure 3: Query evolution comparisons for (a) query execution time (log-scale) and (b) execution time improvement 



compute count. Find MAX user check-in. Compute check-in score 
for each user. 

(b) : EXTRACT user from Twitter log. Compute FOOD- 
SENTIMENT score for each user. 

(c) : CLASSIFY user's tweets from Twitter log as affluent or not. 
GROUPBY affluent users, compute affluent tweet count. Find MAX 
count among all users. COMPUTE an affluent score for each user. 

(d) : Create social network from Twitter log using tweet source 
and dest. GROUPBY user pair in social network, count tweets. 
Find MAX tweets between a user pair. Assign friendship strength 
score to each user pair. 

JOIN (a), (b), and (c) on user_id. Then JOIN with (d) based on 
friend_id . Threshold based on checkin score, food-sentiment score, 
friendship score, and affluent score. 

7.2 Experimental Results 

7.2.1 Query Evolution 

This section presents results for the query evolution scenario, 
where a user's new query can benefit from opportunistic views cre- 
ated during execution of previous versions of the query. In this 
experiment, an analyst (e.g., Ai) executes the first version of his 
query (e.g., AiVi) followed by issuing each subsequent version 
(e.g., AiV2, AiVs, AiVi). Subsequent versions of a query are 
rewritten given the benefit of all opportunistic materialized views 
from previous versions of the query. 



Figure 3(a) shows the performance of the original query plan 
(NO-BFR in graph) and the optimized query plan after rewriting 
with BfRewrite (BFR in graph). Figure 3(b) shows the corre- 
sponding percent improvement in execution time for BfRewrite 
over the baseline original plan resulting from 10% up to 90% im- 
provement, representing an average improvement of 61% and up 
to two orders of magnitude. As a concrete data point, A^va re- 
quires 54 minutes to execute in the baseline system, but only 55 
seconds after BfRewrite finds a suitable rewrite. The amount 
of data manipulated (i.e., read/write/shuffle bytes) closely followed 
the same trend in Figure 3(a). The significant savings in execution 
time for BfRewrite can be attributed to moving much less data, 
and avoiding the costly re-extraction of data from raw logs when- 
ever possible, due to our algorithm's ability to reuse prior compu- 
tation from all previous query versions, i.e., opportunistic material- 
ized views. 

7.2.2 User Evolution 

This section presents results showing the benefit of our methods 
for the user evolution scenario, where a user benefits from oppor- 
tunistic views created during execution of other user's queries. In 
this scenario, a new analyst arrives and executes the first version 
of his query given that all of the other analysts have previously 
executed their queries in the system. Our BfRewrite algorithm 
rewrites AiVi using the opportunistic materialized views from all 
other analyst queries. 
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Figure 4: User Evolution Comparisons for (a) query execution time (log-scale)(b) data moved (c) execution time improvement 



Figure 4(a) shows the query execution time while Figure 4(b) 
shows the data manipulated (read/write/shuffle) for BfRewrite 
and the original plan. The results demonstrate that query execution 
time is always lower with BfRewrite, with a similar trend in the 
amount of data moved. The percentage improvement in execution 
time is given in Figure 4(c) where it can be seen that our system 
results in an improvement of about 50%-90%. 



Table 2: Improvement in execution time of A5V3 as more ana- 
lysts become present in the system 



Analysts in System 


1 


2 


3 


4 


5 


6 
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Improvement 


0% 


73% 


73% 


75% 


89% 


89% 


89% 



As an additional experiment, we show the increasing benefit a 
user may receive each time a new analyst arrives and executes their 
queries. To show this, first we execute A5V3 with no opportunis- 
tic views in the system, to create a baseline execution time. We 
"add" another analyst by executing all four versions of that ana- 
lyst's query, creating new opportunistic views. Then we re-execute 
A5V3 and report the performance improvement for A 5 v 3 over the 
baseline as new analysts are added. We chose as it is a com- 
plex query using all three logs. Table 2 reports the execution time 
improvement as analysts are added, showing an increasing benefit 
as more users are present in the system. The benefit is obviously 
dependent on similarity to other analysts' queries. 

7.2.3 Runtime Comparison o/BfRewrite and DPR 

To evaluate the effectiveness of BfRewrite to prune the search 
space of rewrites, we compare it against the baseline DPR algo- 
rithm that exhaustively searches the space in order to find the same 
minimum-cost rewrite. First we evaluate the algorithms during the 
user evolution experiments, where the only views in the system are 
those created by the other users. This resulted in approximately 
100 views for each user evolution experiment. Next we evaluate 
the algorithm runtime as we vary the number of views in the system 
from 1-1000 to show how the algorithms scale with an increasing 
number of views. 

Figure 5 compares the performance of the BfRe WRITE and the 
dynamic programming algorithm (DPR in graph) during the user 
evolution experiment. While both algorithms find the minimum- 
cost rewrite r* , these figures show that BfRewrite searches 
much less of the space to find the minimum cost rewrite than 
does DPR. Candidate views considered in Figure 5(a) corre- 
sponds to all the existing views (V) and those views created dur- 
ing the rewrite search process (i.e., merged views). The num- 
ber of rewrite attempts corresponds to the candidate views exam- 
ined by REWRITEENUM when searching for a valid rewrite with 
low cost. These results show for many cases it is possible for 
BfRewrite to find r* by considering far fewer candidate views 
as shown in Figure 5(a), and orders of magnitude fewer rewrites at- 
tempted as shown in Figure 5(b), resulting in significant savings in 
algorithm runtime as shown in Figure 5(c). This savings is due 



to BFREWRITE's use of OptCost that enables considering the 
promising candidate views earlier in the search, and exploding the 
space of candidate views incrementally as needed. 

The next experiment evaluates the runtime performance of 
BfRewrite and DPR with an increasing number of materialized 
views in the system. We retained about 9,600 views that were cre- 
ated during the course of design and development of our system, 
including those created during the process of designing the 8 An- 
alyst scenarios. We use the BfRewrite and DPR algorithms to 
find rewrites for one analyst's query (A3V1) given a uniform ran- 
dom sampling of subsets of the existing 9,600 views to show how 
the algorithms scale with the number of materialized views in the 
system. Furthermore, both the algorithms pruned views at a given 
target as follows: a view was pruned if it did not reference the same 
log as the target, if it had a predicate condition preventing it from 
being used in a rewrite for the target, or if it was a view that is iden- 
tical to the target. Identical views were removed because those do 
not help to test the scalability of the algorithms. After pruning, the 
number of materialized views available for each target was slightly 
more than 1000. This experiment was performed many times for 
each algorithm at a given sample size. 
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Figure 6: Runtime of BfRewrite and DPR for varying num- 
ber of MVs in the system 

Figure 6 shows the algorithm runtime for increasing num- 
ber of materialized views in the system. DPR becomes pro- 
hibitively expensive even when 250 MVs are present in the sys- 
tem. BfRewrite on the other hand scales much better than DPR 
and has a runtime under 1000 seconds even when the system has 
1000 views relevant to the given query. While this runtime is not 
trivial, we note that these are complex queries involving UDFs that 
run for thousands of seconds. The amount of time spent to rewrite 
the query plus the execution time of the optimized query is far less 
than the execution time of unoptimized queries. For instance, Fig- 
ure 4(a) reports a query execution time of 45 1 seconds for A$ op- 
timized versus 2134 seconds for unoptimized. Even if the rewrite 
time were 1000 seconds (in this case it is actually 3.1 seconds as 
seen in Figure 5(c)), the end-to-end execution time would be lower 
by 11 minutes or 32%. 

However, additional pruning techniques [2] can be used to re- 
duce the number of views, thereby reducing the algorithm search 
space. We note that BfRewrite will find the minimum cost 
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Figure 5: Algorithm Comparisons for (a) Candidate views considered, (b) Refine attempts, and (c) Runtime (log-scale) 



rewrite in the space it considers. 

7.2.4 Comparison with Caching-based methods 

Here we compare our approach against caching-based methods 
such as ReStore [3] that only reuse identical answers (i.e., existing 
views that require no compensation). Figure 7 shows the query ex- 
ecution time improvements for query evolution of Analyst 1 . The 
results show that both BfRewrite and ReStore are identical for 
A1V2, while ReStore is worse than BfRewrite for A1V2 and 
significantly worse for A1V3. Reusing cached results can clearly 
be effective. However, performance is highly dependent upon the 
presence of identical answers previously generated in the system as 
cache methods lack a robust capability to reuse previous results. 



Table 4: Execution Time Improvement after reclaiming 10% to 
90% of the view storage space 
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Figure 7: Execution time improvement for query evolution us- 
ing BfRewrite and ReStore 



Table 3: Improvement in execution time of user evolution using 
BfRewrite without identical views 



Analyst 


Ai 


A 2 


A3 


A 4 


A 5 


A 6 


A 7 


As 


BfRewrite 


57% 


64% 


83% 


85% 


51% 


96% 


88% 


84% 



To illustrate the robustness of our rewrite algorithm beyond 
caching-based methods, we present an experiment where we prune 
away all identical views from the solution space. Removing all 
identical views represents the worst-case scenario for caching- 
based methods. Such methods will not be able to find any rewrites, 
resulting in 0% improvement. Table 3 reports the percentage im- 
provement using BfRewrite when identical views have been 
pruned from consideration. Our algorithm results in very good 
performance improvement, and these results are comparable to the 
results in Figure 4(c) that represents the same experiment without 
pruning identical views. Table 3 shows a performance drop for 
A5 compared to the results reported in Figure 4(c) for A 5 . This is 
because A 5 had previously benefited from an identical view corre- 
sponding to restaurant similarity which it has to recompute. Note 
that identical views constituted only 7% of the view storage space 
in this experiment. Given that analysts pose different but related 
queries in an evolutionary scenario, any method that relies solely 
on identical matching can have limited benefit. 
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7.2.5 Storage Reclamation 

Since storage is not infinite a reclamation policy is necessary. 
We demonstrate the robustness of our approach using unsophisti- 
cated policies, since reclamation is not the focus of this work. The 
first simple policy randomly drops views to reclaim 10%-90% of 
the view storage space. After each reclamation, we run the A5D3 
query (used in Section 7.2.2) after applying our rewrite algorithm, 
and compare it to a baseline without our rewrite algorithm. Each 
test is repeated twice (different randomizations). Table 4 reports 
the average execution time improvement for A5U3 after reclaiming 
various amounts of storage space. Table 4 shows that our system 
is able to find good rewrites for A5U3 using the remaining views 
available, until most of the storage space is reclaimed. 

Second, Table 3 provides another perspective on how storage 
reclamation can affect the effectiveness of BfRewrite. Specif- 
ically, we observe that BfRewrite is able to find good rewrites 
even if the optimal views for each query are removed. These 
results suggest that our approach is robust to "bad" choices made 
by the reclamation policy, provided some good views remain in 
the system. 

The hardness of designing a good policy is equivalent to the view 
selection problem with a storage constraint. Certainly better meth- 
ods [13] could be applied but we leave this for future work. 

7.2.6 Comparison with DB-X 

To verify that our system produces good rewrites, we com- 
pared our rewrites to those of a widely-used commercial database 
system (DB-X) using data from the TPC-H [19] benchmark. 
DB-X was tuned to its highest optimization level as well as 
set to consider all views for query rewriting. We created two 
materialized views on the LINEITEM table: group-by-count 
on L_ORDERKEY, L_SUPPKEY, and group-by-count on L_- 
ORDERKE Y , L_PARTKEY. To test the rewrites produced by both 
systems, we use three queries on the LINEITEM table: a) groupby 
on L_ORDERKEY, L_SUPPKEY, L_PARTKEY b) group-by on 
L_ORDERKEY with count aggregate, and c) group-by on L_- 
ORDERKEY with count and max aggregate on L_SUPPKEY. The 
resulting rewrites were given to the DB-X query optimizer, which 
provided a cost estimate for each rewrite. 

BfRewrite made use of both materialized views for the first 
query while DB-X did not make use of them which resulted 
in a rewrite that was 3x worse than the rewrite produced by 
BfRewrite. For the second query both BfRewrite and DB- 
X resulted in identical rewrites. For the third query, DB-X did 
not make use of a materialized view, whereas BfRewrite did re- 



suiting in a 5x improvement over DB-X. BfRewrite produces 
rewrites competitive to a state-of-the-art commercial DBMS on 
queries that are much simpler than our setup. Using the DBMS 
to rewrite the complex queries in our workload would most likely 
result in very little improvement. 

8. RELATED WORK 

Query Rewriting Using Views. As noted in Section 6 there has 
been much previous work on rewriting queries using views. Mini- 
Con [16] and [9] both consider rewriting a single query using ex- 
isting materialized views, while [8] extends [9] to consider mul- 
tiple queries. MiniCon extends the bucket-algorithm to be more 
general, and [9] uses a graph approach and applies a form of sub- 
graph matching among queries and views. In these case both the 
queries and views are restricted to the class of conjunctive queries 
(SPJ). Furthermore these methods consider maximally contained 
rewrites as their context is data integration. Our method considers 
SPJGA queries and UDFs and our context is query optimization 
thus we are interested in equivalent rewrites, not maximally con- 
tained rewrites. [16] discusses extensions to cost-based rewriting 
and notes that optimal rewrite for their case is also exponential in 
the number of views present in the system. 

Online Index Selection. Methods such as [10] adapt the phys- 
ical configuration to benefit a dynamically changing workload by 
recommending a set of indexes/views to create or drop. Such ap- 
proaches are not directly applicable to our scenario since material- 
ized views are created simply as artifacts of query execution. View 
selection methods could be applicable during storage reclamation 
to retain only those views that provide maximum benefit within a 
given space budget constraint. 

Reusing Computations in MapReduce. Other methods for opti- 
mizing MapReduce jobs have been introduced such as those that 
support incremental computations [12], sharing computation or 
scans [14], and re-using previous results [3]. As shown in Sec- 
tion 7.2.4, caching-style sharing methods are effective but provide 
limited benefit compared to our approach. 

Multi-query optimization (MQO). The goal of MQO [17] (and 
similar approaches [14]) is to maximize resource sharing, particu- 
larly common intermediate data, by producing a scheduling strat- 
egy for a set of queries in-flight. Our work produces a low-cost 
rewrite rather than a scheduling policy for concurrent query plans. 

9. CONCLUSION 

We presented a method to takes advantage of opportunistic mate- 
rialized views to significantly speedup queries in a large-scale data 
analytics system. With the aid of a UDF model and a lower-bound 
OptCost function we developed in this paper, the BfRewrite 
algorithm produces the optimal rewrite while being work efficient. 
We presented 8 evolutionary queries for realistic scenarios and 
demonstrated dramatic performance improvements with an average 
of 61% and up to two orders of magnitude. 

Because storage is not infinite, future work will address the prob- 
lem of identifying the most beneficial views to retain. We will look 
at view retention strategies from the point of overall system ben- 
efit considering these decisions are affected by view maintenance 
costs, which is an aspect we did not address in this paper. 
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